Exercise #1

library(tidyverse)
library(tidymodels)
library(naniar)
library(vip)
library(lubridate)

theme_set(theme_minimal())

hotels <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-02-11/hotels.csv')

Exercise 1: Setting Up Git and Github in rStudio

Create Repository for Assignment 1. Link: [https://github.com/michaelhetlo/Homework_1_ADR]

Exercise 2: Create Website

a) Link of Website

[https://michaelhelton.netlify.app/]

b) Podcast Reflection

The podcast hosts talked about the importance of having projects you have worked on readily available to you for things like job interviews and networking. They talked about the importance of having expierences to bring up if interviewers talk about struggles or breakthroughs you’ve had on projects in the field. Creating a website to house my old, new and current projects will be a good place to store them for the future when I am looking to apply to jobs, and can have this website as a resume of sorts to show off.

Exercise 3: Modeling Review and Intro to tidymodels

1. a) Without doing any analysis, what are some variables you think might be predictive and why?

Hotel (type), arrival_date_month, booking_changes. Hotel type will be important as individuals traveling for work related purposes I assume will be less likely to cancel reservations, while people on vacation would be more likely to cancel. So I think ‘City Hotel’ will have less cancellations. arrival_date_month might be predictive for the same reason as hotel, trying to figure out who is traveling for work versus pleasure. booking_changes, I assume the higher the changes the less likely someone is to cancel, as the reason they might cancel is less strict because they are booking around it rather than something that comes up that makes it impossible to go.

b) What are some problems that might exist with the data? You might think about how it was collected and who did the collecting.

Redundant variables, ‘is_canceled’ and ‘reservation_status’ for terms of modeling having the extra variable can cause issues. Lack of id’s for each booking is slightly annoying due to possibility compressing multiple reservations into one booking. I am also unsure of how some variables were collected such as previous booking/cancellation data.

c) If we construct a model, what type of conclusions will be able to draw from it?

If families or individuals are more likely to cancel, times of year people are more likely to cancel. I don’t think we will find causation of why things are canceled or not, but we should be able to find some trends of when things are more likely canceled.

2. Create some exploratory plots or table summaries of the data, concentrating most on relationships with the response variable. Keep in mind the response variable is numeric, 0 or 1. You may want to make it categorical (you also may not). Be sure to also examine missing values or other interesting values.

hotels %>%
  group_by(previous_cancellations) %>%
  summarise(total = n())

hotels %>%
  add_n_miss() %>%
  filter(n_miss_all != 0) %>%
  select(children)

hotels %>%
  ggplot(aes(x = customer_type, fill = hotel)) +
  geom_bar()

3. Mutations.

hotels_mod <- hotels %>% 
  mutate(is_canceled = as.factor(is_canceled)) %>% 
  mutate(across(where(is.character), as.factor)) %>% 
  select(-arrival_date_year,
         -reservation_status,
         -reservation_status_date) %>% 
  add_n_miss() %>% 
  filter(n_miss_all == 0) %>% 
  select(-n_miss_all)

set.seed(494)

hotels_split <- initial_split(hotels_mod, prop = .5, strata = 'is_canceled')

hotels_training <- training(hotels_split)
hotels_testing <- testing(hotels_split)

4. Pre-processing

hotels_recipe <- recipe(is_canceled ~ ., 
                       data = hotels_training) %>%
  step_mutate(children = as.factor(ifelse(children == 0,0,1)),
              babies = as.factor(ifelse(babies == 0,0,1)),
              previous_cancellations = as.factor(ifelse(previous_cancellations == 0, 0,1))) %>%
  step_mutate(agent = as.factor(ifelse(agent == 'NULL', 0,1)),
              company = as.factor(ifelse(company == 'NULL', 0,1))) %>%
  step_mutate(country = fct_lump_n(country, n = 5)) %>%
  step_normalize(all_predictors(),-all_nominal()) %>%
  step_dummy(all_nominal(), 
             -all_outcomes())

5. Recipe Building.

In general, why would we want to use LASSO instead of regular logistic regression? (HINT: think about what happens to the coefficients).

We can avoid over fitting the model to the test data, by compressing and using less variable outcomes.

hotels_recipe %>%
  prep(hotels_training) %>%
  juice()

set.seed(494)
hotel_cv <- vfold_cv(hotels_training, v = 5)

hotel_lasso_mod <- 
  logistic_reg(mixture = 1) %>%
  set_args(penalty = tune()) %>% 
  set_engine("glmnet") %>% 
  set_mode("classification")

hotel_lasso_wf <-
  workflow() %>%
  add_recipe(hotels_recipe) %>%
  add_model(hotel_lasso_mod)

hotel_lasso_wf

## ══ Workflow ════════════════════════════════════════════════════════════════════
## Preprocessor: Recipe
## Model: logistic_reg()
## 
## ── Preprocessor ────────────────────────────────────────────────────────────────
## 5 Recipe Steps
## 
## ● step_mutate()
## ● step_mutate()
## ● step_mutate()
## ● step_normalize()
## ● step_dummy()
## 
## ── Model ───────────────────────────────────────────────────────────────────────
## Logistic Regression Model Specification (classification)
## 
## Main Arguments:
##   penalty = tune()
##   mixture = 1
## 
## Computational engine: glmnet

6. Tuning.

penalty_grid <- grid_regular(penalty(), levels = 10)

penalty_grid

hotel_lass_tune <- hotel_lasso_wf %>%
  tune_grid(resamples = hotel_cv,
            grid = penalty_grid)

hotel_lass_tune

hotel_lass_tune %>% 
  collect_metrics() %>% 
  filter(.metric == "accuracy") %>% 
  ggplot(aes(x = penalty, y = mean)) +
  geom_point() +
  geom_line() +
  scale_x_log10(
   breaks = scales::trans_breaks("log10", function(x) 10^x),
   labels = scales::trans_format("log10",scales::math_format(10^.x))) +
  labs(x = "penalty", y = "accuracy")

hotel_lass_tune %>%
  show_best(metric = 'accuracy')

hotel_lass_tune %>%
  show_best()

best_param <- hotel_lass_tune %>% 
  select_best(metric = "accuracy")

best_param

hotel_lasso_final_wf <- hotel_lasso_wf %>% 
  finalize_workflow(best_param)
hotel_lasso_final_wf

## ══ Workflow ════════════════════════════════════════════════════════════════════
## Preprocessor: Recipe
## Model: logistic_reg()
## 
## ── Preprocessor ────────────────────────────────────────────────────────────────
## 5 Recipe Steps
## 
## ● step_mutate()
## ● step_mutate()
## ● step_mutate()
## ● step_normalize()
## ● step_dummy()
## 
## ── Model ───────────────────────────────────────────────────────────────────────
## Logistic Regression Model Specification (classification)
## 
## Main Arguments:
##   penalty = 1e-10
##   mixture = 1
## 
## Computational engine: glmnet

hotel_lasso_final_mod <- hotel_lasso_final_wf %>% 
  fit(data = hotels_training)

hotel_lasso_final_mod %>% 
  pull_workflow_fit() %>% 
  tidy() 

Are there some variables with coefficients of 0?

Yes. ‘arrival_date_month_September’, ‘assigned_room_type_L’, ‘market_segment_Groups’, ‘market_segment_Undefined’, ‘distribution_channel_Undefined’.

7. Testing and Evaluation

hotel_lasso_final_mod %>% 
  pull_workflow_fit() %>% 
  vip()

Which variables show up as the most important? Are you surprised?

‘reserved_room_type_P’, ‘deposit_type_Non.Refund’, ‘assigned_room_type_l’. Why does room type reserved and assigned matter so much?

hotel_lasso_test <- hotel_lasso_final_wf %>% 
  last_fit(hotels_split)

hotel_lasso_test %>% 
  collect_metrics()

How do they compare to the cross-validated metrics?

Test accuracy is .002 higher than my CV metrics on average. And ‘roc_auc’ is about .001 higher.

preds <- collect_predictions(hotel_lasso_test)

preds %>%
  conf_mat(is_canceled, .pred_class)

##           Truth
## Prediction     0     1
##          0 34291  7747
##          1  3292 14363

What is the true positive rate (sensitivity)? What is the true negative rate (specificity)?

True positive rate = 65%. True negative rate = 91%.

preds %>%
  ggplot(aes(x=.pred_1, fill = is_canceled)) + 
  geom_density(alpha = .5, color = NA)

a. What would this graph look like for a model with an accuracy that was close to 1?

There would be very high density peaks at 0 and 1, and very little to no density in the middle.

b. Our predictions are classified as canceled if their predicted probability of canceling is greater than .5. If we wanted to have a high true positive rate, should we make the cutoff for predicted as canceled higher or lower than .5?

Higher cutoff would lead to a higher true positive rate, as the model would only predict cancellations on cases it is very sure of.

c. What happens to the true negative rate if we try to get a higher true positive rate?

The true negative would decrease.

8. Let’s say that this model is going to be applied to bookings 14 days in advance of their arrival at each hotel, and someone who works for the hotel will make a phone call to the person who made the booking. During this phone call, they will try to assure that the person will be keeping their reservation or that they will be canceling in which case they can do that now and still have time to fill the room. How should the hotel go about deciding who to call? How could they measure whether it was worth the effort to do the calling? Can you think of another way they might use the model?

If the hotel is going to call someone, the model should be overwhelmingly certain that the person is likely to cancel. The cutoff would have to be such that the true positive rate is very high, so the hotel isn’t calling people who aren’t likely to cancel. Measure success if the rooms of the people who cancel from that call are getting filled within the 14 days after the call. Determining what travel agents and companies have the most reliable customers, if the contracts they have are worth while for them to keep.

9. How might you go about questioning and evaluating the model in terms of fairness? Are there any questions you would like to ask of the people who collected the data?

Country of origin could be problematic, as well discrimination based on number of children could be an issue. I am also confused on how cancellation and reservation data was collected. If it was based on website booking or travel agents, then those people using those services would be unfairly deemed as less reliable because they have a record. Whereas people who don’t have those online records won’t have the same issues.

Bias and Fairness

Did you hear anything that surprised you?

When responding to the first question from the audience, Dr. Thomas brings up that machine learning has only focused on how to obtain large amounts of data efficiently rather than looking at how the data is obtained and the structure of the data itself. This was an interesting point I hadn’t thought about before, the topic of ensuring test data matches training data. I guess I hadn’t thought about it before because I work with data sets from MLB the most and there isn’t that much difference year to year in outcomes.

Why is it important that we pay attention to bias and fairness when studying data science?

It’s important to pay attention to bias and fairness because that work data scientists do can have large impacts on people’s lives, for example models that predict recidivism send can keep people in jail based on factors that have nothing to do with the individual due to large amounts of bias built into the process. The outcomes of these models can prevent freedom from people. On a smaller scale we have to make sure our model does what we want it do. We want the model to be fair across and points in the data so we must make sure there isn’t a large bias towards a subsection of our data set.

Is there a type of bias Dr. Thomas discussed that was new to you? Can you think about places you have seen these types of biases?

Representation bias. I saw a lot of these types of biases when working over last summer. I was working on predicting account viability, the database had something like 10% that were not viable accounts that I was trying to predict would not be viable. I made a model and was impressed with myself that I got it up to 90 some odd percent accuracy. Until I went back and looked at the confusion matrix and saw that it was predicting almost all accounts to be viable, so it was really bad at predicting non-viable accounts. I eventually got around this by constructing a dataset that was 50-50 viable to non-viable, and quickly saw that my model in actual terms sucked.